Motion picture Travelling Matte
In the motion picture industry, the travelling matte process photographs a subject before a backing of uniform color and luminance. A matte signal is generated from the color differences between the colored backing and the subject. This matte signal permits removal of the colored backing, and the subsequent insertion of a different background.
The matte signal is simply a measure of the visibility of the background in the photographed scene. It can also be expressed as the contribution of the background to each pixel in the image frame. Since the background cannot be seen behind an opaque foreground subject, its contribution (matte) is zero. In all areas of the background not obscured by the subject, the background is fully visible, and the matte is therefore 1.0.
The matte is a fraction between 0.0 and 1.0, at the edge of moving or out-of-focus subjects where a semitransparent blur occurs, and for semitransparent and translucent objects such as glassware, smoke, or fog. Semitransparent subject areas contain a mix of the subject color and the color of the background since both occupy the same area. It is the background contribution that must be removed from these semitransparent areas. The matte signal is a measure of that contribution, and is used for its removal.
The above process has been well developed and is literally flawless for most subjects. See U.S. Pat. Nos. 4,100,569, 4,344,085; 4,625,231, 5,032,901, 5,343,252, and 5,424,781.
In both motion pictures and graphic arts, it is often necessary to select a foreground subject in an existing scene and remove its background, so that a different background may be inserted. The well developed matte equations for uniform colored backings do not apply to a background consisting of multiple colors and objects such as people, streets, trees, buildings and blue sky.
The development of a workable background extraction method for such complex backgrounds is described in U.S. patent applications Ser. Nos. 09/008,270 and 09/141,703. This extraction method employs a computer, cursor and monitor for manipulation of picture elements.
Application Ser. No. 09/008,270 is briefly reviewed here, since reference to this method will be made in the disclosure of the present invention. A boundary line is drawn on the background just outside the subject, on the original image, to isolate the background from the foreground subject. The isolated background is assigned a matte level of 1.0 since it is unobscured by the subject. A second boundary line is drawn just inside the subject to isolate the opaque subject, and this isolated subject area is assigned a matte level of zero. Between these two boundary lines is the transition area between subject and background. Semitransparent subject edges due to subject motion, or lack of sharp focus, are included in the transition area.
A background reference frame is computer generated by extrapolation and interpolation of the RGB levels of background pixels into and through the area occupied by the subject. The background reference frame is what one would have seen in the absence of the subject. This reference frame need be relatively accurate only through the transition area between the two outlines.
A foreground subject reference frame is generated by extrapolating the subject RGB levels near the subject edges to stretch the subject out through and beyond the outlined transition area.
Each pixel in the original image frame has a corresponding address in each of the two reference frames. These three frames each have separate colors for the same pixel address. These three colors can be shown as three points in a three dimensional color space as noted in FIG. 2 of the referenced application Ser. No. 09/008,270. The position of the image pixel color with respect to the background and foreground reference frame colors determines the contribution (matte level) of the background to the image pixel.
The above method requires the drawing of a line on each side of the transition between subject and background. While this method worked very well for most subjects and backgrounds, consider a complex subject such as a leafy tree covering much of the image frame. The background shows blue sky, white clouds and mountains. Portions of the leaves are illuminated by the sun, and the sky peeks through between clumps of leaves.
There is no expedient way to separate foreground and background by drawing boundary lines. These kinds of busy subjects and backgrounds led to the invention of an alternative method, described below, in which the matte is generated from multiple conditional and Candidate mattes.